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Abstract 


In  many  experimental  situations  the  pertinent  inferences  are 
made  on  the  basis  of  orthogonal  contrasts  among  the  treatment  means 
(as  in  2n  factorial  experiments).  In  this  setting  a  particularly 
useful  form  of  inference  is  one  involving  multiple  comparisons .  The 
present  paper  describes  situations  in  which  such  inferences  are  mean¬ 
ingful,  gives  examples  of  their  use,  and  provides  an  extensive  set  of 
tables  of  constants  needed  to  implement  such  multiple  comparison  procedures. 
The  procedures  can  also  be  used  for  statistically  legitimate  -Kdat a  snooping" 
(in  the  sense  of  Scheffe  (1959),  p.  80)  to  help  decide  which  contrasts 
within  a  specified  set  warrant  further  study . 

KEY  WORDS :  Multiple  comparisons,  orthogonal  contrasts,  joint  confidence 

4 

intervals,  experimentwise  error  rates,  Studentized  maximum  modulus, 
simultaneous  inference. 


1 .  INTRODUCTION 


The  results  of  many  planned  experiments  can  be  analyzed  in  terms 
of  meaningful  orthogonal  contrasts  among  treatment  means.  This  is  the 
situation,  for  example,  in  2n  factorial  experiments,  and  in  experiments 
with  quantitative  factors  for  which  orthogonal  polynomials  (Fisher  and 
Yates  (1938-1963);  Davies  (1978),  Appendix  8C)  are  employed  to  fit  a 
regression  curve  or  surface;  Cochran  and  Cox  (1957),  Sections  3. 4-3. 5, 
discuss  orthogonal  contrasts  in  some  detail.  Ackermann  (1979)  proved  a 
general  formula  to  calculate  the  values  of  orthogonal  polynomials  for 
the  case  of  nonequidistant  levels  and  unequal  numbers  of  observations. 

For  hypothesis  testing  the  orthogonality  of  the  treatment  contrasts 

makes  it  possible  to  partition  the  sum  of  squares  for  treatments  into 

a  set  of  one-degree-of-freedom  sums  of  squares  which  add  up  to  the  sum 

of  squares  for  treatments;  under  normality  each  individual  sum  of  squares 

.  '  2 

associated  with  one  of  the  treatment  contrasts  is  distributed  as  x 
with  one  degree  of  freedom,  independently  of  all  of  the  other  individual 
sums  of  squares.  The  orthogonality  of  the  treatment  contrasts  also  guarantees 
that  their  usual  best  linear  unbiased  estimators  are  normally  and  independently 
distributed. 

If  only  one  contrast  is  of  interest,  then  Student's  t  can  be  used  for 
hypothesis  testing  or  for  interval  estimation.  However,  in  most  experiments 
there  is  more  than  one  contrast  which  is  of  interest.  The  problem  then 
becomes  somewhat  more  complicated  for  joint  inferences  are  now  involved,  and 
the  experimenter  may  desire  to  control  the  experimentwise  error  rate.  Thus, 
e.g.,  when  two  or  more  hypotheses  are  each  tested  separately  at  level  of 
significance  ct  using  (say)  Student's  t,  the  experimentwise  error  rate 
(which  controls  the  probability  of  at  least  one  false  positive)  is  greater 


-2- 


than  a.  To  help  compensate  for  this  effect  some  experimenters  use, 
for  each  test,  a  common  value  of  a  which  is  smaller  than  the  one  that 
would  typically  be  employed  if  only  one  hypothesis  were  being  tested; 
others  use  critical  values  based  on  Bonferoni  inequalities.  (The 
problem  is  further  complicated  by  the  fact  that  the  individual  tests 
are  not  independent  since  each  usually  employs  the  same  residual 
mean  square  as  the  estimator  of  the  underlying  variance.)  Analogous 
problems  arise  when  two  or  more  orthogonal  contrasts  are  to  be  estimated 
jointly  using  one-sided  or  two-sided  confidence  intervals. 

In  the  present  paper  we  shall  mainly  consider  joint  two-sided 
confidence  interval  estimation  of  orthogonal  contrasts;  the  joint  con¬ 
fidence  interval  approach  is  usually  more  relevant  than  joint  hypothesis 
testing.  In  any  case,  joint  interval  estimation  can  easily  be  re-formulated 
as  joint  hypothesis  testing.  We  shall  describe  situations  in  which  the 
joint  interval  estimation  approach  would  appear  to  be  appropriate,  and 
show  how  to  make  exact  confidence  statements  concerning  the  joint  interval 
estimates  of  the  orthogonal  contrasts  which  are  of  interest.  In  Section  2 
we  provide  an  extensive  set  of  tables  of  constants  which  are  needed 
to  implement  the  procedure.  The  constants  are  based  on  a  special  case 
of  the  multivariate  Student's  t  of  Dunnett  and  Sobel  (1954)  and  Cornish 
(1954).  The  theory  underlying  these  tables  is  given  in  Section  2  along 
with  a  description  of  the  method  underlying  their  construction,  and 
comments  on  the  accuracy  of  the  entries  therein.  Section  3  describes 
some  examples  of  the  application  of  the  tables. 
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2.  DISTRIBUTION  THEORY  AND  TABLES 


2.1  Distribution  theory 


We  assume  that  the  random  variables  Y^  (1  <  i  _<  k;  j  =  1,2,...,N) 

are  independent  and  normally  distributed  with  E{Y..}  =  u.  and 

13  1  k 

2  2  r 

Var{Y..}  =  a  ,  the  {y.}  and  o  being  unknown.  Define  0  -  )  c  .u. 

i]  l  m  .  mi  l 

(1  <  m  <  p)  where  the  c  .  (1  <  i  <.  k,  1  <  m  <.  p)  are  specified  constants 
”  k  ml 

such  that  £  c  .  =  0  (1  <  m  <  p).  The  9  represent  a  family  of  p 

i=l 

contrasts  among  the  ik  ;  we  suppose  that  the  experimenter  is  interested 
in  obtaining  two-sided  interval  estimates  of  the  9^  with  specified  joint 
confidence  coefficient  1-os.  Such  interval  estimators  with  the  required 
joint  confidence  coefficient  are  given  by 


■{v 


0  (  8  ±  h/  J  c.s/N>  (1  <  m  <  p) 

m  m  mi  j  = 


where  0 


K.  W  *  2 

=  y  c  .y. ;  y.  -  J  Y. ./N,  s  is 
•  S  mi  i’  l  ii 

i=l  1=1 


the  usual  unbiased  estimate 


2  2  2  2 
of  a  based  on  v  d.f.  (vS  /a  is  distributed  as  xv*  independently 

of  9  =  (§,...,§  )),  and  h  is  a  constant  chosen  to  satisfy 

'VI  p 


P{  n  B  }  =  1-a. 
,  m 
m=l 


In  order  to  determine  the  value  of  h  to  be  used,  we  note  that  £ 
has  a  p-variate  normal  distribution  where  the  elements  of  £ 

are  given  by 


2 

=  (  y  c  .c  . )o  /N  ( 1  <  m  ,m  <  p) , 
.  ,  m,  i  m  _  i  i  i 


Then  the  distribution  of  £  =  (T^,...,T^)  where 


is  a  central  p-variate  Student  t-distribution  with  v  d.f.  Its  joint 
density  function  (see  Dunnett  and  Sobel  (1954)  or  Cornish  (1954))  is  given 
by 

2 

(2.1) 


f  (  t,  ,  •  •  •  ,t  i 

V  1  p 


V 


A|1/2r[(vtp)/2]  r  |  1 


(vTT)p/2r(v/2) 


r  +  v  i  .  i  ai  j  rj 

L  j  si  i  =1  ■ilJ2  -1 


■  t. 

^1  ]2_ 


where  the  {a_.  ^  }  are  the  elements  of  £  =  £  Let  |^|  =  (  |T^ | , . . . , jl^ j ) 
where  £  has  the  joint  density  function  (2.1),  and  let 


Gv(h;p,£)  =  P{|Tj  <  h  (1  <  m  <  p)} 

h  h 

=  /  •••  /  fv(t  t^)dt...dt  (2.2) 

-h  -h  ^  ^ 

00 

- / 

0 

-  m-i, . . .  ,p 


J  •  •  •  J  <t»(x. 


-hs<x  <hs 


1” 


,x. 


•’VtXj. 


. .  ,dx 


gv(s)ds 


where  4>(x^, . . .  jX^;]^)  is  the  standard  p-variate  normal  probability  density 

function  (pdf)  with  correlation  matrix  P  =  {p  },  and  g  (s)  is  the 

'v  m^m^  v 

pdf  of  S.  If  o  =0  (m  /  m„,  1  <*  m  ,m  <_  p)  then  the  multiple 

ni^ni^  i  z  i  z 

integral  within  the  square  brackets  of  (2.2)  can  be  expressed  as  an  iterated 
integral  (thus  simplifying  the  problem  of  evaluating  it  numerically),  and 
(2.2)  reduces  to 


H  (h;p,p)  =  / 
0 


/_ 

Vu- 


hs+p1/2yA  _ 


p) 


1/2 


-hs+p1/ 


-P ) 


1/2 


xd$(y)dy 


g  ( s ) ds  (2.3) 

\> 


where  $(  • )  is  the  standard  normal  cdf . 


The  constants  h  =  h^(p,p,a)  satisfying  H^(h;p,p)  =  l-o  have  been 

tabulated  to  two  decimal  places  by  Dunnett  (1964)  for  p  =  0.5; 

p  =  2(1)12,15,20;  a  =  0.05,0.01;  v  =  5(1)20,24,30,40,60,120,*.  Hahn 

and  Hendrickson  (1971)  tabulated  h  to  three  decimal  places  for 

p  =  0.0, 0.2, 0.4, 0.5;  p  =  1(1)6(2)12,15,20;  a  =  0.10,0.05,0.01; 

v  =  3(1)12,15(5)30,40,60.  Other  earlier  tabulations  are  cited  in  Hahn 

2 

and  Hendrickson.  Krishnaiah  and  Armitage  (1970)  tabulated  h  to  two 

decimal  places  for  p  =  0.1(0. 1)0. 9;  p  =  1(1)10;  a  =  0.05,0.01;  v  =  5(1)35. 

For  the  special  case  of  orthogonal  contrasts  with  which  we  are  con- 
k 

cerned  here,  we  have  J  c  .c  .  =  0  (m,  /  m„,  1  <.  m, ,m„  <  p);  hence 

.  m^i  1  2  —  1  2  ■■ 

p  =  0  and  (2.3)  becomes 


00 

H^(h;p ,0)  =  /  [$(hs)  -  $(-hs)]Pg^(s)ds. 


(2.4) 


Since  p  =  0  we  see  that  Hahn  and  Hendrickson's  Table  1  is  applicable. 


f  c2.S2/N  i! 


In  this  case  the  statistic  M(p,v)  =  max  0  -0  /  )  c  .S  /N  is 

'mm'  . - ,  mi 
l<m<p  i=l 

known  as  the  Studentized  maximum  modulus,  and  P(M(p,v)  <  h)  =  H^(h;p,0). 

Pillai  and  Ramachandran  (1954)  had  earlier  tabulated  h  for  p  =  0  to  two 

decimal  places  for  p  =  2(1)8;  a  =  0.05;  v  =  5(5)20,24,30,60,120,*.  Stoline 

and  Ury  (1979)  have  tabulated  h  to  three  decimal  places  for  p  =  k(k-l)/2, 

k  =  3(1)20;  a  =  0.2,0.1,0.05,0.01;  v  =  5,7,10,12(4)24,30,40,60,120,*; 

in  a  later  paper  (Ury,  Stoline  and  Mitchell  (1980))  these  tables  were 

extended  to  cover  k  =  20(2)50(5)80,90,100;  a  =  0.2,0.1,0.05,0.01; 

v  =  20(1)40(2)60(5)120,240,480,*.  (The  constants  tabulated  by  Stoline 

et  al.  were  to  be  used  for  joint  two-sided  interval  estimates  for  the 

k(k-l)/2  pairwise  contrasts  u.  -u.  between  the  k  population  means. 

11  i2 

Such  contrasts  are  not  orthogonal,  but  as  a  consequence  of  an  inequality  of 


Sidak  (1967),  the  use  of  constants  h  determined  for  the  case  p  =  0 
results  in  intervals  which  are  conservative ,  i.e.,  they  achieve  a  joint 
confidence  coefficient  which  exceeds  the  nominal  1-ct  at  the  cost  of 
having  intervals  which  are  somewhat  broader  than  they  need  be.)  Earlier, 
Games  (1977),  employing  Sidak's  (1967)  multiplicative  inequality,  computed 
conservative  constants ,  specifically  for  the  problem  of  multiple  compari¬ 
sons  for  non -orthogonal  contrasts,  his  tables  give  upper  bounds  to  h 
to  three  decimal  places  for  p  =  2(1)10(5)50;  a  -  0.20,0.10,0.05,0.01; 
v  =  2'(1)30,40,60, 120,“.  (We  mention  that  Chen  (1979)  tabulated  percentage 
points  associated  with  random  variables  arising  from  a  multivariate  Student 
t-distribution  (2.1)  with  zero  correlations.  However,  his  tables  are  not 
related  to  ours;  they  are  used,  for  example,  to  find  an  interval  estimate 
of  max{y  ,...,y  }  when  the  estimates  y.  (1  <  i  <  k)  are  based  on 

x  K  1 

sample  sizes  which  are  not  necessarily  all  equal.) 

« 

In  applications  involving  multiple  comparisons  for  orthogonal  con¬ 
trasts,  our  tables  provide  the  constants  needed  to  obtain  two-sided 
interval  estimates  of  p  contrasts  with  joint  confidence  coefficient 
exactly  equal  to  the  nominal  value  of  1-ct.  The  tables  give  h  =  h  (p,p,a) 
to  five  significant  figures  for  p  =  0;  p  =  2(1)31;  a  =  0.2,0.1,0.05,0.01, 
v  =  2(1)30(5)60,120,240,“. 

2 . 2  Construction  of  the  tables 

In  order  to  construct  the  tables  it  was  necessary  to  obtain  the 
solution  in  h  to  Hv(h;p,0)  =  1-a  where  H^(h;p,0)  is  given  by  (2.4). 

To  evaluate  the  infinite  integral  for  finite  values  of  v,  a  96-point 
Legendre  quadrature  formula  was  used  (see  Abromowitz  and  Stegun  (1964), 
p,  919);  for  v  =  ®  we  have  s  =  1  and  the  infinite  integral  disappears. 
This  method  of  calculation  is  not  appropriate  for  v  =  1  (since  the  pdf 


•  ...  ...j***^a*4*± 
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g  (s)  is  then  infinite  at  s  =  0),  and  no  h-values  were  comDutad  for 
v 

v  =  1.  The  general  computer  program  which  was  written  for  arbitrary  o 
and  h  was  first  used  for  p  -  1/2,  p  =  2,  and  the  probabilities  obtained 
were  then  checked  against  those  given  in  Table  1  of  Dunnett  3nd  Sobei 
(1954)  which  had  been  computed  using  an  exact  series  expansion;  agreement 
was  found  to  the  five  decimal  places  given  in  each.  The  h-values  were 
also  checked  against  those  given  in  Hahn  and  Hendrickson's  Table  1  and 
in  Stoline  and  Ury  (1979),  and  agreed  to  the  three  decimal  places  given 
in  these  tables.  Our  h-values  are  believed  to  be  correct  to  the  four 
decimal  places  given. 

3.  ILLUSTRATIONS  OF  USES  OF  THE  TABLES 

In  this  section  we  consider  several  types  of  planned  experiments  in 
which  the  pertinent  inferences  are?  made  on  the  basis  of  orthogonal  con¬ 
trasts  among  the  treatment  means.  We  shall  point  out  some  of  the  issues 
involved,  and  indicate  how  the  multiple  comparisons  procedures  used  with 
the  appropriate  constants  in  our  tables  can  control  the  experimentwise 
error  rates  for  such  inferences. 

3.1  Experiments  involving  a_  single  Qualitative  factor 
Example  1:  A.  5 -level  experiment 

3ennett  and  Franklin  (1954),  Section  7.34,  consider  an  experiment 

involving  five  different  methods  of  analyzing  the  concentration  of  iron 

in  a  standard  solution.  Two  methods  included  agitation  and  three  methods 

did  not;  four  analyses  were  made  with  each  method.  The  orthogonal 

contrasts  under  consideration  (see  their  Table  7.9)  were  (c  ,,c  „,...,  c  . ) 

mi  ttu  me 

(3, 3, -2, -2, -2) ,  (1,-1, 0,0,0),  (0,0, 2, -1,-1) ,  (0,0,0, 1,-1)  for  m  =  i,2,3,u, 
respectively.  Here  p  =  4,  v  =  15  and  from  our  Tables  1  and  2  we  find 
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that  h^g(4,0,a)  =  3.6082  and  2.3051  for  a  =  0.01  and  0.05, 
respectively.  The  joint  confidence  intervals  with  an  experimentwise 
error  rate  of  a  are  thus  given  by 


C  ~  / - 5 —  "N 

ei  ~  h15(4*°>a)  *30sV4 


§2  ±  his(4,0,a)  /2s  /4 


§3  ±  h^2(4,0,a)  /6s  /4 


S4  ±  h15C4,0,a)  /^4 


(3.1) 


As  pointed  out  in  the  text,  the  breakdown  into  single  degrees  of 
freedom  in  any  particular  situation  will  be  dictated  by  the  characteristics 
of  the  experiment.  Two  cases  frequently  used  for  4-ievel  experiments 
are  (cnl’Gm2’cm3,Cm4)  *  >  (1,-1, 1,-1),  (1,-1, -1,1)  and 

lcml’cm2,cm3,cm4)  =  (1,'1,0’0)’  U.1.-2.0),  (1,1,1, -3)  for  a  =  1,2,3, 
respectively .  _ 

If  the  number  of  analyses  made  with  each  method  were  not  ail  equal, 
some  (and  possibly  all)  of  the  contrasts  under  consideration  would  not 
be  orthogonal .  Then  the  use  of  our  h-values  yields  a  confidence 
coefficient  greater  than  1-a.  (See  §idak  (1967),  equation  (8).) 

3.2  2n  factorial  experiments 

4 

Example  2:  A  2_  experiment 

Cochran  and  Cox  (1957),  Section  5.24a,  analyze  the  yields  obtained 

4 

in  a  2  experiment  with  four  fertilizers  (m  »  manure,  n  *  nitrogen. 


p  *  phosphorus,  k  «  potassium),  each  at  two  levels,  conducted  in  four 
randomized  complete  blocks;  the  experiment  was  carried  out  to  study  the 
effects  of  these  fertilizers  on  the  yield  of  grass.  The  64  yields  per 


plot  (total  over  5  harvests,  km.  per  3-meter  row)  were  used  to  compute 


the  effect  means  which  were  given  as  M  *  13.3,  N  =  21.3,  ?  =  5.5, 
k  =  24.1,  MN  =  3.2,  MP  *  -0.3,  MK  *  -7.5,  WP  =  3.5,  N'K  =  10.9,  ?K  =  3.2, 
MNP  =  -1.4,  MNK  =  -3.5,  MPK  *  0.3,  NPK  =  0.5,  MNPK  =  -1.6.  The 
estimated  standard  error  of  an  effect  mean  was  computed  to  be 
/ s^/2n  ^r  =  t/(90.5)/4(4)  s  2.38  where  s'  *  90.5  is  the  error  mean 
square  based  on  v  *  45  d.f.,  r  *  4  is  the  number  of  replications 
of  each  treatment  combination,  and  n  *  4  is  the  number  of  factors. 

For  v  =  45  the  Student  t-values  are  2.690  and  2.014  for  a  =  0.01  and 
a  -  0.05,  respectively,  'he  ef'act  means  -  t*s(2.38)  are  exhibited 
in  Columns  2  and  3  of  Tab’.  »  £  for  a  =  0.01  and  a  -  0.05,  respectively. 
Cochran  and  Cox  give  analogous  information  in  the  lower  half  of  their 
Table  5.1a  where  they  indicate  effects  that  are  statistically  significant 
at  the  1%  (**)  and  5%(*)  levels ,  namely  ( M,N,K,MK,NK,MNK)  and  (P), 
respectively. 

Tc  control  the  exper imentwise  error  rate,  the  corresponding  h-values 
from  our  Tables  1  and  2  are  h^ ^( 15 ,0 , 0 . 01)  =  3.6503  and  h^,( 15 , 0  , 0 . 05 > 
3.0803  for  a  *  3.01  and  a  ~  0.05,  respectively.  The  effect  means 
o  hu  g(  15 ,0  ,.'i '  (2.38)  are  exhibited  in  Columns  ■+  and  5  of  Table  5  for 


3  *  0.01 

and 

3  * 

0.05,  respectively.  We  note 

from  Columns  3  ar.c  a 

that  MK 

and 

MNK 

are  now  "significant''  at  the 

5^  (rather  than  at  the 

1%  level) 

and 

that 

P  is  not  significant  at  the 

5%  level.  Thus  for  th 

experiment,  controlling  the  exper imer. tw  i  s  e  error  rate  has  reduced  the 
number  of  statistically  significant  result?. 

The  experimenter  must  decide  whether  the  per  contrast  cr  tr.e  experim 
wise  error  rate  is  more  pertinent  in  his  particular  experiment.  It  the 
extreme  case  of  e  single  significant  contrast,  the  experimenter  using 
per  contrast  error  rates  would  be  left  feeling  unsure  whether  the  effect 
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Table  5 

4  1/ 

A  2  complete  factorial  experiment— 


(p  =15,  v  =  45) 


Confidence  interval  on  effect  mean—' 


Effect 

Error  rate  a 
per  contrast 

Error  rate  a 
per  experiment 

a  =  0.01 

tj5  =  2-690 

a  -  0.05 

<5  *  2'014 

a  =  0.01 

hac  =  3.6503 
45 

a  =  0.05 

h“5  =  3.0803 

M 

18.016.4(**) 

18 . 0±4 . 8( *) 

18 . 018 . 7( **) 

18 . 0i7 . 3( *) 

N 

21 . 3±6 . 4( **) 

21 . 3±4 . 8(  *) 

21 . 318 . 7( **) 

21 . 3i7 . 3( * ) 

P 

5. 5+6. 4 

5 . 5+4 . 8( *) 

5.518.7 

5.517.3 

K 

24.1±6.4(**) 

24 . 1±4 . 8( * ) 

24 . 118 . 7( **) 

24.117. 3(*) 

MN 

3.2±6.4 

3. 2+4. 8 

3.218.7 

3.217.3 

MP 

-  0 . 3±6 .4 

-  0 . 3±4 . 8 

-  0.318.7 

-  0.317.3 

MK 

-  7 . 516 . 4( **) 

-  7.5+4.8(*) 

-  7.518.7 

-  7.5+7. 3(*) 

NP 

3. 5±6 .4 

3 . 5±4 . 8 

3~.  518. 7 

3.517.3 

NK 

10 . 916 .4( **) 

10.9±4.8(*) 

10 . 918 . 7( **) 

10.917. 3(*) 

PK 

3. 2+6-4 

3. 2+4. 8 

3.218.7 

3.2+7.3 

MNP 

-  1.4+6. 4 

-  1.4+4. 8 

-  1.418.7 

-  1.417.3 

MNK 

-  8.5+6.4(**) 

-  8. 5+4. 3(A) 

-  8.518.7 

-  8 . 5+  7 . 3  ( * ) 

MPK 

0 . 8±6 . 4 

0 . 8±4 . 8 

0.818.7 

0.817.3 

NPK 

0.5±6.4 

0.514.8 

0.519.7 

0.517.3 

MNPK 

-  1 . 6±6 . 4 

-  1.614.8 

-  1.618.7 

-  1.617.3 

—  Cochran  and  Cox  (1957),  Table  5.1a. 

— ^Standard  error  of  effect  mean  is  /s2/2n_2r  =  /( S0.5)/4( 4)  =  2.38. 
The  entry  in  each  cell  in  the  body  of  the  table  is  either  (effect 
mean)  *  t®g(2.38)  or  (effect  mean)  i  h“5(2.38).  The  intervals 
indicated  by  (**)  or  (*)  do  not  cover  zero. 
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was  a  real  one,  given  that  the  experiment  has  provided  multiple  oppor¬ 
tunities  for  one  of  them  to  be  significant.  On  the  other  hand,  in  an 
experiment  with  several  significant  contrasts  as  in  the  present  example, 
strict  use  of  the  experimentwise  error  rate  procedure  makes  it  more 
difficult  for  the  smaller  effects  to  be  declared  significant  after 
the  larger  ones  have  been  identified. 

We  have  assumed  in  the  above  analysis  that  a_  priori  the  experimenter 
was  interested  in  controlling  the  experimentwise  error  for  all  15 
contrasts.  If,  a  priori,  he  had  been  interested  in  only  the  4  main 
effects  and  the  6  two-factor  interactions  (and  he  had  made  that  decision 
without  being  influenced  by  the  data)  then  p  =  10,  v  =  u5,  and  the 
appropriate  h-value  for  a  =  0.01  and  a  =  0.05  would  be  3.5149  ana  1 

2.9346,  respectively,  now  MK  is  still  significant  at  the  5%  level  and 

t 

P  is  still  not  significant  at  the  5%  level,  while  the  status  of  MNK 
(as  well  as  MNP,  MPK,  NPK  and  MNPK)  in  terms  of  possible  significance 
would  be  unknown.  If,  after  looking  at  the  data ,  the  experimenter  decided 
that  only  M,N,K,NK  and  MNK  were  of  interest,  then  he  still  must  use 
the  original  factor  h^^X  15 ,0 ,a)  in  reporting  his  final  results. 

Effectively,  what  he  has  done  here  is  "data  snooping"  in  the  sense  of 
Scheffe  (1959),  d.  30,  and  he  must  pay  for  that  privilege  by  using  the 
larger  h-value  if  he  desires  to  make  statistically  legitimate  confidence 
statements  with  experimentwise  control  over  the  error  rate.  In  this 
situation  the  inference  must  be  limited  to  a  particular  set  of  orthogonal 
contrasts  specified  in  advance. 

It  thus  is  clear  that  in  2n  complete  factorial  experiments  where 
n  is  "large,"  it  is  to  the  experimenter's  advantage  if  he  can  specify 
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a  priori  which  contrasts  are  and/or  are  not  of  interest;  analogous 
considerations  hold  for  fractional  factorial  experiments.  In  certain 
types  of  experiments  it  is  easy  to  identify  certain  contrasts  which 
are  not  of  interest.  We  consider  such  a  problem  in  Example  3. 

3 

Example  3 :  A  2__  experiment  with  two  classification  factors 

Consider  a  3-factor  experiment,  each  factor  at  two  levels,  where 

the  factors  are  diet  (.Diet  1  vs.  Diet  2),  sex  (male  vs.  female),  and 

age  (old  vs.  young).  The  purpose  of  the  experiment  is  to  study  the 

effect  of  change  in  diet  on  gain  in  weight.  Here  the  treatment  factor 

of  interest  is  diet  while  sex  and  age  are  classification  variables . 

Thus,  denoting  the  main  effect  of  diet,  sex  and  age  by  A,  3  and  C, 

respectively,  and  analogously  for  their  interactions,  the  experimenter 

would  be  interested  in  the  p  =  4  orthogonal  contrasts  associated  with 

A,  AB,  AC  and  ABC  rather  than  in  all  7  orthogonal  contrasts.  Similarly, 

4 

in  a  2  experiment  two  of  which  are  classification  factors,  the  experi¬ 
menter  would  be  interested  in  at.  most  the  p  =  12  orthogonal  contrasts 
A,  B,  AB,  AC,  AD,  ACD,  BC,  BD,  BCD,  ABC,  ABD,  ABCD.  See  Cox  (1958), 
Examples  6.3  and  6.4,  for  a  discussion  of  treatment  factors  and 
classification  factors. 

3.3  3n  factorial  experiments ,  all  factors  quantitative 

3 

Example  4 :  A  3_  experiment 

3 

Davies  (1978),  pp.  332-336,  reports  the  results  of  a  3  experiment, 
each  factor  quantitative  and  equally  spaced,  all  treatment  combinations 
replicated  twice.  The  variable  under  study  is  the  yield  of  a  chemical 
process,  the  three  factors  being:  i)  C,  the  concentration  of  an 


' >  . 


inorganic  material  (A)  in  the  free  water  present  in  the  reaction  mixture, 

ii)  V,  the  volume  of  free  water  present  in  the  reaction  mixture,  and 

iii)  N,  the  amount  of  a  second  inorganic  material  (B)  in  the  reaction 
mixture.  Each  factor  was  studied  at  three  equally  spaced  levels,  and 
two  replications  of  each  of  the  27  treatment  combinations  was  obtained. 

A  quadratic  response  surface  was  fit  to  the  data  using  orthogonal  poly¬ 
nomials,  and  the  total  d.f.  for  treatments  was  partitioned  into  18 

individual  d.f.  associated  with  the  6  main  effects  (C  ,C  ,V  ,V  ,Nr  ,N  ) 

L  L  u  ^ 

and  the  12  two-factor  interactions  (C^*V.  ,  C^xy, ,  CT*VA,  C^xV^,  CT  *N.  , 


Q’  Q  Q’ 


C„*Nr  >  C  xN  i  C  xN  ,  V  4  ,  V  *H  ,  V  xN  V  xN);  the  remaining  3  degrees 

^  x  LL  xx 

of  freedom  representing  the  three-factor  interactions  were  pooled.  Here 

the  subscripts  L  and  Q  rep'resent  linear-  and  quadratic,  respectively. 

There  were  27  d.f.  associated  with  the  error  mean  square. 

For  v  =  27  the  Student  t-vaiues  are  2.771  and  2.052,  for  a  -  2.01 

and  oi  =  0.05,  respectively.  Using  these  values  (actually  the  corresponding 
1  2 

F97  =  (t07)  values  were  used)  Davies  reported  the  3  effects 
(D.  ,  CA,  Vr  .  V  ,  CT  xV.  ,  C  xy,  ,  D  <i;.  )  as  beine  statisticallv 

significant  at  the  It  level  and  the  .  effects  (. N ,  C,  xM.  )  significant 
at  the  10%  level.  ( Note :  We  are  reporting  the  results  here  as  tests  of 
significance  to  conform  with  Davies,  but  we  would  have  preferred  to  present 
our  results  as  interval  estimates  as  in  our  Table  5.) 


To  control  the  experimentwise  error  rate  the  corresponding  h-values 
from  our  Tables  1,  2  and  3  are  h-_( 13,0,0 .01)  =  3.3989  ,  h„_( 13 ,0,0. 05)  = 

Z  I  £.  . 

3.2560,  and  h^7(  18 ,0 ,0 . 10 )  =  2.9582  for  a  =  0.01,  cc  =  0.05  and 
a  =  0.10,  respectively.  Thus,  if  the  experimenter  wished  to  control  the 
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experimentwise  error  rate  for  the  18  orthogonal  contrasts  of  interest 

he  would  assert  that  the  3  effects  (C  ,  C  ,  C  *V  )  are  significant 

«-»  Q  U  Li 

at  the  1%  level,  the  3  effects  (V,  ,  V  ,  C  *N  )  are  significant  at  the 

x  y  L 

5%  level,  the  one  effect  (C^xVr )  is  significant  at  the  10%  level,  and 
the  remaining  11  effects  are  not  significant  at  the  10%  level.  (The 
experimenter  had  decided  a  priori  that  the  3  effects  associated  with  the 
three-factor  interactions  were  not  of  interest,  and  hence  this  total  sum 
of  squares  based  on  8  d.f.  was  not  partitioned  into  the  3  individual 
one  d.f.  sums  of  squares  associated  with  each  of  the  remaining  relevant 
orthogonal  contrasts.) 

Thus  the  same  considerations  arise  in  the  analysis  of  this  experimen 

U. 

as  arose  in  the  analysis  of  the  2  experiment  of  Example  2.  And  the 
same  caveats  hold  here  as  well. 

3.4  An  application  to  biological  assay 

The  purpose  of  biological  assay  is  to  estimate  the  potency  ratio, 
c,  of  two  biological  preparations  which  have  dose-response  curves  which 
can  be  represented  by  the  same  form  of  regression  function  and  which 
differ  only  in  the  factor  p  in  the  dose  scale.  (Vie  use  the  symbol 
p  here  as  in  Finney  (1978),  p.  41.)  A  common  situation  is  one  in  which 
the  response  scale  is  linear  in  log  dose.  In  this  case  parallel  straight 
lines  can  be  fit  to  the  two  sets  of  data;  an  estimate  of  log  p  is  then 
given  by  the  horizontal  distance  between  them.  A  useful  experimental 
design  for  such  situations  is  the  so-called  symmetric  (k,k)-poir.t  design 
in  which  k  dose  levels  equally  spaced  on  a  log  scale  are  used  for  each 
preparation  (the  k  levels  being  different  for  the  two  preparations) 
with  n  observations  being  taken  at  each  of  the  2k  design  points; 
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the  usual  values  for  k  are  2,  3  or  u.  See  Finney  (1973),  p.  135. 

In  analyzing  the  data  from  a  (k,k)  bioassay,  the  sum  of  squares 
between  treatments  has  2k-l  d.f.  which  can  be  separated  into  2k-l 
meaningful  orthogonal  components.  See  Finney  (1978).  pp.  135-139,  for 
an  example  with  k  =  3  where  the  5  orthogonal  contrasts  are  denoted  by  L 
(preparations),  (average  linear  regression),  (parallelism), 

L0  (average  quadratic  regression),  (difference  between  quadratics). 

The  first  two  enter  into  the  calculation  of  the  estimated  relative  potency- 
while  the  remaining  three  are  used  to  test  the  validity  of  the  assay. 
However,  significance  tests  at  level  a  =  0.35  for  each  of  these  three 

O 

will  result  in  an  error  rate  for  the  assay  approaching  1  -  (1-a)'"'  =  3.1-. 

If  it  is  desired  to  -control  the  experimentwise  error  rate  at  a  specified 
value  a,  then  the  constants  h  tabulated  in  our  paper  can  be  used. 

To  apply  our  constants  to  Finney's  example,  the  largest  of  the 
three  mean  squares  for  the  validity  contrasts,  which  here  is  12  having 
a  value  of  0.001606  (see  Finney's  table  5.2.2),  is  expressed  as  a  ratio 

to  the  error  mean  square  based  on  30  d.f.;  this  ratio  then  is  compared  for 

o 

a  =  0.05  with  (h^C  3,0  ,0 . 05) )  =  (2.5224)“  =  6.36  instead  of  refer'  ing 

’12  2 

it  to  tables  of  F“Q  =  (t.^)  =  (2.042)  =4.17.  In  this  experiment  the  ' 

ratio  is  only  0.52,  a  clearly  non-sigr.if icant  result. 

In  Finney's  description,  the  contrast  is  also  considered  as  a 

P 

test  of  a  type  of  assay  validity,  and  as  such  it  could  be  included  with 
the  other  three  to  form  a  set  cf  four  simultaneous  tests  involving  orthogona 
contrasts.  Actually’,  provides  a  measure  of  how  successful  the  experi- 

menter  has  been  in  choosing  comparable  dose  levels  of  the  two  preparations, 
and  a  significant  value  provides  a  signal  to  the  experimenter  that  he  might 


be  comparing  the  two  preparations  at  different  portions  of  the  dose- 


response  curve  rather  than  necessarily  invalidating  the  assay.  Thus  it 
might  be  preferable  to  consider  it  separately  from  the  other  three 
validity  contrasts,  as  illustrated  in  the  preceding  paragraph . 

Similar  considerations  apply  with  more  than  three  dose  levels  of 

each  preparation.  For  example,  in  a  (4,4)  bioassay  there  would  be  5 

orthogonal  contrasts  in  addition  to  L  and  L  .  If  linearity  is 

P 

assumed  then  these  provide  5  separate  tests  for  assay  validity  which  can 
be  tested  by  using  h  (5,0,a)  from  our  tables  in  order  to  achieve  an 
experimentwise  error  a  for  the  validity  tests.  On  the  other  hand,  if 
a  quadratic  dose  response  curve  is  assumed,  two  of  them  enter  into  the 
calculation  of  the  estimated  potency,  as  described  by  Finney  (1973), 
p.  122,  leaving  a  set  of  3  orthogonal  contrasts  to  test  the  validity  of  • 
the  bioassay. 
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